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ABSTRACT 
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Generally, students were referred to ESS because of poor academi.c 
performance. Students* goals were heavily influenced by their teachers, yet 
they adopted the goals as their own. ESS classrooms engaged in student-led 
activities. Students received individualized instruction as needed and had 
the opportunity to have concepts not mastered retaught to them. All 
stakeholders agreed that the program helped students increase their academic 
achievement. Parents and students reported improved study skills and 
increased motivation to learn. Major program strengths included early 
targeting of students ,. dedicated staff, student transportation, collaboration 
between teachers and coordinators, flexible scheduling, low teacher-student 
ratios, and individualized instruction. Improvements could be made in areas 
of student transportation, staff development, expanded services in terms of 
hours and/or subjects, and a reduction of the student-teacher ratio, all of 
which require increased funding. Twelve recommendations are discussed. (TD) 
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INTRODUCTION 



Background 

The Extended School Services (ESS) program was established in 1990 as part of the 
Kentucky Education Reform Act (KERA). Designed specifically to address the needs of 
Kentucky's at-risk student population, ESS is an aggressive, proactive program for addressing 
academic problems before they become ingrained (Nesselrodt & Schaffer, 2000b). The ESS 
program extends the school day, week, or year for students at risk of academic failure, providing 
them with additional instructional time to help them meet academic goals. Rather than being an 
“add-on” or “stand-alone” program, ESS is designed to be an integral part of each school's 
regular academic program, thus ensuring that students receive instructional assistance in core 
content subjects in which they are performing poorly. 

The major emphases of the statewide ESS program are to (1) sustain students’ present 
level of performance to prevent them from falling behind; (2) provide extended programming for 
students who have been retained or are at risk of (a) being retained in a class or grade or 
(b) failing to graduate on time without assistance; and (3) close the achievement gap of low- 
performing students so they will perform successfully in the program appropriate to their age. 
According to publications from the Division of Extended Learning of the Kentucky Department 
of Education (KDE), nearly every school provides such services; thus, about 1,450 schools have 
some type of ESS program (AEL, 2001; Quality Education Data, 1998). See the Kentucky ESS 
Web site for more detailed information (http://www.kde.state.ky.us/osle/extend/ESS/default.asp). 



Past Evaluations 

Prior to this evaluation, three major within-state (internal) evaluations of the ESS 
program have been completed — one by the University of Kentucky in 1991, one by the KDE in 
1993, and one by the Joint Center for the Study of Educational Policy at the University of 
Kentucky and the University of Louisville in 1998. In 1999, the Kentucky Commissioner of 
Education called for an external evaluation, i.e., by an agency(ies) outside of Kentucky. This 
evaluation was to be piloted in the spring of 2000 and conducted during the 2000-01 academic 
year (Nesselrodt & Schaffer, 2000b). 

The KDE approved a plan submitted by Drs. Pamela Nesselrodt and Eugene Schaffer (of 
Dickinson College, Pennsylvania, and the University of Maryland, respectively), which focused 
on four major categories related to the ESS program: (1) identification, referral, and assignment 
of services; (2) profiles of students receiving services; (3) profiles of ESS programs; and (4) out- 
comes of the programs (Nesselrodt & Schaffer, 2000a). Nesselrodt and Schaffer completed a 
pilot-test evaluation of the ESS program in the spring of 2000 that resulted in two reports — one 
on the design, testing, and refinement of instruments and another on the refinement and 
finalization of research questions and methodology. 
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Current Evaluation 

^ In the fall of 2001, KDE contracted with a partnership of AEL and Western Kentucky 

University for a comprehensive evaluation of the ESS program during the 2001-02 school year. 
All learnings from the pilot-test evaluation were incorporated into AEL's evaluation design. 
Fifteen evaluation questions were assembled into five major topics: (1) identification, referral, 
and assignment of services; (2) profiles of students receiving services; (3) profiles of ESS 
programs and their implementation patterns; (4) services to students placed at risk; and (5) ESS 
implementation patterns and outcomes. 

AEL's evaluation of the ESS program utilized two major components — statewide surveys 
and site visits. These components were broken dovm into five main phases: state\vide surveys, 
training session for site visits, fall/winter site visits, summer visits, and data analyses. The 
surveys were administered to the district and school ESS coordinators in the fall of 2001. See 
Figure 1 for a graphic portrayal of AEL’s evaluation of the ESS program. 

The major purposes for the site school visits in the fall/winter of 2001-02 and the summer 
of 2002 were to provide intensive, extensive ESS program data that would be generalizable, 
valid, and reliable to Kentucky programs statewide and to corroborate findings from the 
statewide surveys previously administered to district and school ESS coordinators. The site 
visits replicated most of the procedures and data collection instruments utilized in the pilot-test 
evaluation. A pair of trained data collectors made two to three day visits to a sample of 24 
schools with ESS programs (18 during the fall/winter of 2001-02 and 6 in the summer of 2002) 
to collect both qualitative and quantitative data from a variety of ESS stakeholder groups.* 
See Table 1 for descriptive information about the sites selected for visits. 

The site visit data collection involved classroom and ESS session observations; 
interviews with ESS teachers, ESS students, ESS parents, the school ESS coordinator, and the 
ESS district coordinator; surveys of non-ESS teachers, ESS teachers, ESS students, and ESS 
parents; a school and program description form; and written documentation such as the school's 
consolidated plan and needs assessment, as well as descriptions/policies of the ESS program. 
AEL added two new instruments — the Innovation Component Configuration Map — to generate 
patterns of implementation across ESS programs, and the AEL Continuous School Improvement 
Questionnaire, to measure the extent to which a school faculty is committed to continuous 
improvement. 



* A two-stage sampling process was implemented to identify the 24 schools. In the first stage, KDE staff established a 
pool of 48 schools through a six-step process that reviewed student achievement data, percentage of students eligible 
for fi'ee or reduced-price meals, overall academic student index, ethnicity, school-level performance indicators such as 
novice-level readers and dropout rates, comparisons of subsets of student scores within schools, and geographic and 
demographic representations. AEL completed the second stage by securing Johnson locale codes (National Center for 
Education Statistics, 2001) and published enrollment figures (Quality Education Data, 1998) for each of the 48 schools. 
Using a combination of building level, geography, urbanicity, and enrollment, AEL staff selected the 1 8 schools for the 
fall/winter 2001-02 visits. AEL staff collaborated by telephone with KDE staff to identify which of the 18 schools 
would be revisited during the summer of 2002, based on geography, building level, and general representativeness of 
Kentucky ESS programs. 
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Two Statewide Surveys 

• District ESS Coordinators 

• School ESS Coordinators 



► 



Report on 
Both 
Surveys 



Sample of 48 Kentucky Schools 




AEL CSIQ 


• 24 Elementary Schools 




Administered 


• 12 Middle Schools 


W 


to 48 Schools 


• 12 High Schools 







FallAVinter Site Visits 
• 18 School Sites with ESS 

• Surveys: 

• ESS Teacher 

• Non-ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Interviews: 

• District ESS Coordinator 

• School ESS Coordinator 

• ESS Teacher 

• Parent of ESS Student 

• ESS Student 

• Classroom Observations: 

• Classroom Observation Form 

• QAIT Classroom Rating 

• Classroom Environmental and 
Resources Checklist 

• School and Program Description Form 

• ICCM Form 



► 



Final 
Report of 
the ESS 
Evaluation 



Summer Site Visits 

• 6 of 1 8 Sites Above 

• Same Surveys, Interviews, 

Observations, and Forms less the 
District ESS Coordinator Interview 
and the Non-ESS Teacher Survey 




Figure 1 : Graphic Portrayal of the Evaluation 
of the Kentucky Extended School Services Program 
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Table 1: Descriptive Information on School Sites Selected for Visits 
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METHODS 



The first task for this comprehensive evaluation was to convert all of the instruments used 
in the Nesselrodt and Schaffer pilot test (2000a, 2000b) to a machine-scannable format 
(excluding the interview protocols) and to make improvements based on lessons learned from the 
pilot test and suggestions from KDE staff These instruments included six surveys (district 
coordinator, school coordinator, ESS teacher, non-ESS teacher, ESS student, and parent of ESS 
student); five interview protocols (district coordinator, school coordinator, ESS teacher, ESS 
student, and parent of ESS student); the Special Strategies Observation System (SSOS), which 
included three forms related to classroom observation, quality of instruction, and environmental 
resources of the classroom; and a school and program description form. AEL staff also selected 
and prepared in scannable format two other data collection instruments. First was the AEL 
Continuous School Improvement Questionnaire (AEL CSIQ). Second was the Innovation 
Component Configuration Map for Extended School Services (ICCM). This first task was 
completed by August 2001 and all instruments were submitted to the AEL Institutional Review 
Board (IRB) for approval. A few minor changes were made in response to IRB suggestions and 
all instruments were approved for use in the evaluation. 



Statewide Surveys 



For a complete report on the instrumentation, data collection and analysis, and findings 
for the statewide coordinator surveys, see Perceptions of Kentucky ’s Extended School Services 
Program by District and School Coordinators (Cowley & Meehan, 2001). 



Training Session for Site Visits 



An experienced data collector was identified to conduct the training session for site visits. 
This consultant was hired in September and was especially instrumental in helping revise the 
SSOS instruments. With the data collection instruments approved for use and a trainer on board, 
the next steps turned to designing the training session, developing the training manual, and hiring 
data collectors. Data collectors included six AEL staff, four experienced Kentucky 
educators/consultants, and two West Virginia consultants. 

A training manual was developed for data collectors to use during and after the training 
session. This manual contained copies of each instrument to be used in the evaluation along with 
instructions for administration. It also included an agenda, a participant list, an overview of the 
ESS project and evaluation, sections for each of the major types of data collection, procedural 
information (random selection information, student consent forms, site visit procedures, 
materials checklist), a calendar for scheduling site visits, and an evaluation form for the training 
session. An e-mail listserv was developed so that data collectors could interact quickly and 
easily with one another and with AEL evaluation staff during the evaluation project. 
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The training session took place on October 3-5, 2001. Twelve individuals completed the 
three-day training. The bulk of the training time was devoted to the three instruments in the 
SSOS. Discussion of codes, practice with classroom videotapes, and instructions proceeded on 
schedule. One important aspect of the training was the inclusion of actual practice observations 
conducted in classrooms at a nearby high school. After each live classroom observation during 
the second and third days of training, a debriefing session was conducted to discuss coding 
questions and concerns. The training also dealt with other instruments such as interviews and 
surveys. The AEL director of evaluation conducted the sessions on the ICCM and AEL CSIQ. 

Several wrap-up activities were completed at the conclusion of the training session. One 
was an evaluative activity in which participants coded a criterion tape of classroom behavior 
previously selected by the trainer, who reviewed each completed SSOS and verified that all 
participants met or exceeded the established level of competency in terms of coding 
specifications. A second wrap-up activity was making tentative pairings of data collectors and 
schools, based on interests, schedules, and geographical proximity. The third and final wrap-up 
activity was the completion of an evaluation form by all participants. 



AEL CSIQ Administration 



Instrumentation. The AEL Continuous School Improvement Questionnaire (AEL 
CSIQ) is a 60-item instrument that measures the extent to which a school’s faculty members are 
committed to continuous improvement. The 60 items are each rated on a Likert-type scale of 
1 to 6 (Not present to Present to a high degree). These items comprise six scales: (1) learning 
culture, (2) school/family/community connections, (3) shared leadership, (4) shared goals for 
learning, (5) purposeful student assessment, and (6) effective teaching. This instrument was used 
to discern the connection between faculty members’ commitment to continuous improvement and 
implementation of their ESS programs. 

Data collection. During the second week of October, AEL staff drafted a letter for the 
Kentucky Commissioner of Education’s signature, soliciting cooperation from the 48 schools in 
this step of the evaluation. AEL staff then assembled kits containing the cover letter, the 
instruments, and a postage-paid return envelope; these kits were mailed on October 15. AEL 
staff called all non-responding schools and spoke with the principal or school coordinator to 
solicit information regarding the status of the instrument completion. In late January, KDE staff 
contacted the few remaining schools. By the end of February, 47 of the 48 schools had returned 
completed forms, for a return rate of 98%. 

Data analysis. AEL staff scanned the surveys using Remark optical scanning software, 
cleaned the data files, and exported them to a standard software program (SPSS) for statistical 
analyses. Analysis was conducted at the scale level, and appropriate descriptive statistics were 
generated. Analyses were generated for each individual school and for aggregated building-level 
profiles for elementary, middle, and high schools. AEL CSIQ data also were analyzed to 
determine whether significant differences existed by ESS implementation patterns. Descriptive 
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statistics and one-way Analysis of Variance (ANOVAs) were utilized at the scale level for this 
analysis.* 

Validity and reliability. The validity and stability reliability of this instrument was 
established in prior research conducted by AEL staff (Meehan, Cowley, Craig, Balow, & 
Childers, 2002). As a measure of the internal consistency reliability of the instrument for this 
administration, Cronbach alpha values were computed for the 10 items within each of the six 
scales, as well as for all of the items in the instrument. For this set of scores, alpha values ranged 
from .89 for the learning culture scale to .94 for effective teaching; the overall value for all items 
was .98. 



Site Visits 



To begin the process of site visits, AEL staff drafted another letter for the 
Commissioner’s signature, announcing to the 18 schools (6 elementary, 6 middle, and 6 high 
schools) that they had been selected. KDE staff mailed the letters on October 26, 2001, and data 
collectors began contacting schools to schedule their site visits. By the second week of 
November, all of the visits were scheduled for completion at the end of February 2002; most data 
collection teams were scheduled for three site visits. Of the original 1 8 schools selected for the 
fall/winter site visits, one was eventually replaced with an alternate due to a scheduling conflict. 

On April 25, a conference call involving AEL and KDE staff was held to select the 
summer site visit schools and two alternates. The decision was made to revisit 6 of the original 
18 schools (2 elementary, 1 middle, and 3 high schools). Such a strategy would provide data 
from the schools’ regular after-school ESS programs and their summer programs. KDE staff 
contacted an administrator at each school and secured their participation. 

All six visits were conducted during June 2002. In terms of methodology, the only 
differences in the summer site visits were the exclusion of the non-ESS teacher survey, the 
decision not to re-interview the district administrator, an increase in the targeted number of 
parent and student interviews, and an increase in the number of ESS students observed. As well, 
a few of the interview questions for parents, students, ESS teachers, and school coordinators 
were eliminated because they were not relevant to the summer programs. 

At the conclusion of each site visit, data collectors returned materials to AEL. Each set 
of school materials was logged in and checked for completeness. Data collectors were notified 
of any missing materials; completed materials were filed and stored by school. All signed 
consent forms were sent to AEL’s Administrative Services Office for archival purposes. 
Interview notes were submitted to AEL support staff for typing before qualitative analysis began. 



*Random sampling was not utilized in this evaluation, given the original sample pool of 48 schools provided by 
KDE. Therefore the assumption for the use of ANOVAs and t tests does not apply in a classical sense. However, 
they are informative for comparing within-group variance to between-group variance for some of the measures used 
in this evaluation. 
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The following sections describe the instrumentation, data collection, data analysis, and 
reliability information for each of the four major techniques utilized during the site visits: 
surveys, interview protocols, classroom observations, and other instruments. 



Surveys 

Instrumentation. Four survey instruments were developed, tested, and utilized in the 
ESS pilot test and employed during the site visits of this evaluation. The four target groups 
included ESS teachers, non-ESS teachers, ESS students, and parents of ESS students. Each 
survey is briefly described below. 

• ESS teacher: This survey contained 24 questions. The initial 18 items were selected- 
response and focused on demographics, attributes of the ESS program, recruitment 
procedures, teacher hiring and staff development, communication strategies, major 
outcomes, forces that help or hinder implementation, and overall effectiveness of the 
program. Six open-ended items asked for information related to strengths, weaknesses, 
and recommendations for improvements. This survey was administered at school to the 
full population of ESS teachers. 

• Non-ESS teacher: This survey contained 19 questions: 13 selected-response and 6 open- 
ended items, all similar to those on the ESS teacher survey. This survey was 
administered at school to the full population of teachers not engaged in the ESS program. 

• Parent of ESS student: This survey contained 1 1 questions. The initial 7 items were 
selected-response and focused on their children’s performance in ESS. Four open-ended 
items focused on best features or problems of the program and an explanation of why 
their children would or would not participate the following year. This survey was 
administered to parents of all students participating in the ESS program who had returned 
a signed parental consent form. One copy of the survey was sent home with the student and 
was completed by one parent individually or by both collaboratively. 

• ESS student: This survey contained 16 questions. Thirteen of the items were selected- 
response and focused on demographics, subjects studied in ESS, and a series of questions 
about academic behaviors and attitudes with which students agreed or disagreed. Three 
open-ended items focused on what students liked best about the ESS program and what 
changes should be made to the program. This survey was administered at school to all 
students participating in the ESS program who had returned a signed parental consent 
form authorizing their participation in evaluation activities. 

Data collection. During each site visit, the data collectors worked with the school 
coordinator to administer the four surveys. Surveys were generally distributed during the first 
day of the visit, and returned by the last day. There was wide variance among the participation 
rate of students, due to the fluctuating percentages of signed consent forms. 
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Data analyses. After all of the fall/winter site visits were completed, AEL staff designed 
data entry templates using Remark optical scaiming software. Surveys were scanned and then 
exported to SPSS for statistical analysis, including descriptive statistics. Comments were typed 
into the SPSS data files. Summer site visit data were scaimed as they became available. 
Individual school files were merged into one master file by type of survey before analyses began. 
A breakdown of the number of surveys completed during the fall/winter and summer site visits is 
provided in Table 2. 

Table 2: Number of Site Visit Evaluation Survey Respondents by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


ESS teachers 


175 


50 


225 


Non-ESS teachers 


297 


NA 


297 


ESS students 


775 


445 


1,220 


Parents of ESS students 


360 


216 


576 


Totals 


1,607 


711 


2,318 



Validity and reliability. These surveys were developed, tested, and utilized in the pilot 
test by Nesselrodt and Schaffer (2000a, 2000b). The instruments possess face and content 
validity and have proven their utility in a prior administration. To assess the degree of internal 
consistency reliability, Cronbach alpha coefficients were computed for this administration of the 
selected-response items on the four surveys, excluding demographic-type items and, for the 
parent survey, one particular multiple-response item. This administration of the ESS teacher 
survey resulted in a coefficient of .60; the non-ESS teacher survey, a .53; the parent survey, a 
.53; and the student survey, a .70. 



Interview Protocols 

Instrumentation. Five interview protocols were developed, tested, and utilized in the 
pilot test and employed in this evaluation. The five target groups included district administrators 
(coordinators), school administrators (coordinators), ESS teachers, ESS students, and parents of 
ESS students. Each protocol is described briefly below. 

• District and school coordinators: These protocols each contained 20 questions and 
focused on aspects such as the history of ESS programs in the district/school; a 
description of the services provided under the current program; how students are 
identified and referred to the program; recruitment and staff development for 
administrators and teachers; communication among teaching staff, students, and parents; 
processes for setting goals and monitoring student progress; and assessment and exit 
procedures. 

• ESS teacher: This protocol contained 16 questions and focused on aspects such as a 
description of the current ESS program, recruitment and staff development practices. 
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communications with students and parents, curriculum and methodologies used in regular 
and ESS classrooms, and major strengths and weaknesses of the current ESS program. 

• Parent of ESS student: This interview protocol contained 1 1 questions and focused on 
perceived effectiveness of the program, areas of potential change, and perceived growth 
or success of their children. 

• ESS student: This protocol contained 6 main questions, with multiple sub-items within 4 
of the main questions and focused on perceived effectiveness of the program, areas of 
potential change, and perceived growth or success. 

Data collection. Arrangements were made to interview the district coordinator at either 
the district office or the school during each site visit. With the exception of students, all 
interviews were conducted on an individual basis. Extensive interview notes were taken, which 
were later transcribed for analysis purposes. Students, parents, and teachers were randomly 
selected for interviewing. For students, a random selection was made using the signed parental 
consent forms received by the school coordinator. 

Data analyses. By the end of the summer site visits, all interviews had been completed. 
A breakdown of the number of interviews completed during the fall/winter and summer site 
visits is provided in Table 3. Typed interview responses were analyzed by questions, first for 
categories then for emergent themes. Next, data displays were developed for each question. 

Table 3; Number of Site Visit Evaluation Interview Participants by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


District coordinators 


15* 


0 


15 


School coordinators 




6 


23 


ESS teachers 


72 


26 


98 


ESS students 


74 


35 


109 


Parents of ESS students 


36 


13 


49 


Totals 


214 


80 


294 



*Three of the district coordinators had two schools within the 18 schools selected for site visits. 
**One of the school coordinators was unavailable due to an injury suffered the day before the 
site visit. 

Validity and reliability. These protocols were developed, tested, and utilized in the pilot 
test by Nesselrodt and Schaffer (2000a, 2000b). These protocols possess face and content 
validity and have proven their utility in a prior administration. For reliability, interviewers’ use 
of protocols at the training session and during the site visits established a satisfactory level of 
agreement. 
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Classroom Observation Instruments 

Instrumentation. For the collection of regular classroom and ESS session data, three 
instruments were selected and employed in the pilot test and used in this evaluation. All three 
were developed and employed in prior research and evaluation studies, refined by AEL and KDE 
staff, and converted to a scannable format. The three instruments comprise the Special Strategies 
Observation System (SSOS), which is designed for use in a variety of settings to systematically 
collect data on essential elements of classroom behavior related to instruction, management, and 
context. The SSOS is a viable instrument for school effectiveness research due to its strong 
grounding in the current literature on effective teaching and its utilization of a variety of 
methodologies. This combination of instruments generates low-, moderate-, and high-inference 
data; this triangulation of information further documents the veracity of the data collected. Each 
instrument that makes up the SSOS is described below. 

• Classroom Observation Form (COF): The COF is a combination observation system 
that is best described as a category system, with low inference items, and includes 
multiple coding procedures (Nesselrodt & Schaffer, 1993; Sullivan & Meehan, 1983). It 
is based on the Classroom Activity Record designed by Everston and Burry (1989) and 
the Stallings Observation System (Stallings, 1980). The top page of the COF collects 
typical demographic information, such as the school, observer, date, number of adults and 
students in class, subject being observed, and type of class (ESS or regular). The 
observations occur over 56 minutes, during which the observer switches between coding 
the entire classroom and focusing on a single student previously selected. Each of seven 
pages corresponds to eight minutes of class time. The first minute per page — the 
“snapshot” — looks at student engagement (i.e., the number of students on task, off task, 
out of the room, or waiting) and grouping strategies (i.e., whether clustered in teacher, 
aide, or student groups and type of involvement, such as working alone, management, 
interaction, or socialization). The remaining seven minutes per page focus specifically on 
the target student, and include coding one of 27 discrete activities for each minute. 

• QAIT assessment of classroom: This instrument is best described as a moderate and 
high-inference, simple coding, rating device. QAIT stands for Quality of Instruction, 
Appropriate Level of Instruction, Incentive, and Use of Time. Fitting on two sheets, it 
contains 40 items grouped under the four major categories. Each item uses a Likert-type 
rating scale of 1 to 5 {Unlike this class to Like this class). This instrument was to be 
completed at the end of each observation session. 

• Classroom environment and resources: The Classroom Environment and Resources 

Checklist (CERC) is a low-inference, simple coding, sign system. Printed on the front of 
one sheet, it contains 12 classroom attributes that are coded either as present or not 
present, such as adequate lighting, use of multi-racial materials, posted assignments, etc. 
Next, 18 classroom resource items, such as textbooks, computers, and worksheets are 
listed; observers indicate whether such resources are visible or not. If they are, observers 
indicate whether they are used during the observation. This instrument also was to be 
completed at the end of each observation session. 




16 



12 



Data collection. Data collectors utilized these forms during the school site visits. The 
classroom observation segments were completed during the observation; QAIT and CERC forms 
were completed as soon after the observation as feasible. Only ESS observations were 
completed during the summer visits, as regular school was not in session. 

Data analyses. After the fall/winter site visits were completed, AEL staff designed data 
entry templates using Remark scanning software. SSOS data were scanned by school; data files 
were then cleaned and exported to SPSS for statistical analyses. School files were merged into 
one master file before analyses began. A total of 1 93 student observations was completed during 
the fall/winter school visits and 20 were completed during the summer visits for a grand total of 
213 observations. 

Classroom observation data were averaged across the number of eight-minute intervals 
per each observation. Percentages of time for the classroom snapshots and target student 
activities were calculated for both the regular classroom and ESS sessions. Data were analyzed 
using the 27 individual categories and by grouping these into four main categories: teacher-led, 
student-led, management/organization, and off-task. Student engagement data and time spent by 
the target student in the four main instructional categories also were analyzed by ESS 
implementation patterns. 

QAIT data were analyzed by creating four main subscales composed of the 40 individual 
items. Descriptive statistics were used to describe results for both regular classrooms and ESS 
sessions. Further, independent t tests were conducted to determine if a statistically significant 
difference existed between the two types of classes for each of four categories: quality of 
instruction, appropriate level of instruction, incentive, and use of time. 

CERC data were analyzed by calculating frequency percentages showing whether the 
classroom attributes were present in the regular classroom and ESS sessions. As well, frequency 
percentages were calculated to show whether various classroom resources were visible and used 
during the observations. 

Validity and reliability. These instruments were tested and utilized in the pilot test by 
Nesselrodt and Schaffer (2000a, 2000b). Thus, these instruments possess face and content 
validity and have proven their utility in prior research. A high degree of inter-rater reliability 
was achieved among the data collectors, given that every participant passed at or above the 85% 
criterion of the COF coding assessment held at the conclusion of the training session. To assess 
the degree of internal consistency reliability, Cronbach alpha coefficients were computed for this 
administration of the COF and QAIT instruments, excluding demographic-type items; this 
procedure was not appropriate for the CERC instrument, given its lack of variance in response 
options of either selected or not selected. For the COF instrument, this administration of the 
grouping strategy items resulted in a coefficient of .54; for the student engagement items, a .82. 
For the QAIT instrument, this administration of all items resulted in a coefficient of .94; by 
subscale, the coefficients were .91 for quality of instruction, .74 for appropriate level of 
instruction, .88 for incentives, and .80 for use of time. 
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Other Instruments 

Instrumentation. The School and Program Description Form was used in the pilot test 
and then refined by KDE and AEL staff for use in this evaluation. The form is machine 
scannable and fits on one sheet. It contains 15 items, 13 of which are demographic in nature, i.e., 
school characteristics, student enrollment, number of students and teachers involved in the ESS 
program, hours of operation, etc. The two open-ended items ask for a description of the major 
components of the ESS program and any unique characteristics of the school or community. 

The Innovation Component Configuration Map for Extended School Services (ICCM) 
was developed by the Kentucky Institute for Education Research (KIER, n.d.) in the mid-1990s. 
The ICCM is based on the Concems-Based Adoption Model (CBAM) originated and developed 
by Gene Hall, Shirley Hord, and others (Hall & Hord, 1987; Hord, Rutherford, Huling- Austin, & 
Hall, 1987). Basically, the ICCM is a map depicting the 15 major components of the ESS 
program in the three broad areas of student eligibility, school level program design, and district- 
wide ESS program planning. For each major component, there are three or four possible 
implementation variations (coded as 1 to 3 or 4, as appropriate), which were summed to create a 
total implementation score. The appropriate implementation levels for each component are 
determined through an analysis of all relevant data gathered during the site visit. When 
completed, a picture of the ESS component configurations was established based on the 
implementation scores. When the ICCMs for all the site visits were completed, then a study of 
their various patterns of implementation was conducted; patterns were then used in conjunction 
with other variables to determine whether statistically significant differences occurred among the 
identified implementation patterns. 

Data collection. The School and Program Description Forms were to be completed and 
gathered during each of the 24 site visits. However, only 22 of these forms were returned to 
AEL for analysis. An ICCM form was completed by the data collectors for each school site visit 
and returned to AEL. 

Data analyses. A data entry template was developed for each instrument using Remark 
scanning software. After the instruments were scanned, the files were cleaned and then exported 
to SPSS for analysis. Descriptive statistics were generated for the School and Program 
Description Form items; the two open-ended items were qualitatively analyzed by common 
themes. Descriptive statistics were generated for the 15 ESS components on the ICCM. 
Further, AEL staff met to visually examine the ICCMs to detect patterns of implementation. 
Based on this discussion, the only patterns emerging were based on the summed score for each 
school, which could range from 15 to 47 (the greater the score, the greater the degree of 
implementation). These patterns were used to correlate with a number of other school-level 
variables, including proficiency and accountability levels, attendance and retention rates, number 
of parent volunteer hours, average years of teaching experience, expenditures per student, and 
number of drug/weapon/assault incidents. These patterns also were used to determine 
statistically significant differences for student engagement and instructional activity categories 
from the SSOS, selected key items from ESS coordinator and teacher surveys, and the AEL 
CSIQ scales. 



ERIC 




14 






Validity and reliability. The School and Program Description Form was developed, 
tested, and utilized in the pilot test by Nesselrodt and Schaffer (2000a, 2000b). The ICCM is 
based on solid research endeavors such as the Concems-Based Adoption Model. Thus, these 
instruments possess face and content validity and have proven their utility in a prior 
administration. To assess the degree of internal consistency reliability, Cronbach alpha 
coefficients were computed for this administration of the ICCM; this procedure was not 
appropriate for the School and Program Description Form, given its strictly descriptive nature. 
This administration of all items in the ICCM instrument resulted in a coefficient of .82; by the 
three broad areas, the coefficients were .78 for student eligibility, .74 for school-level program 
design, and .44 for district-wide ESS program planning. 
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FINDINGS 



Findings are presented as comprehensive summaries of data points that are directly 
aligned with the five broad evaluation topics and their related subquestions. 



Evaluation Topic One: Identification, Referral, and Assignment of Services 

a. Why do students participate in ESS? 

Nearly all (95%) of the 151 district coordinators who responded to the district 
coordinator survey indicated that most students were referred to the program because they were 
in danger of failing and needed to improve their academic performance. In addition, 90% of the 
district coordinators interviewed (n = 18) indicated that the program was intended to help 
students succeed, or improve academically. In addition, 73% or more of both ESS (n = 225) and 
non-ESS (n = 297) teachers surveyed indicated that their students received ESS services to 
improve academic achievement and because their students were in danger of failing. Additional 
reasons that students received ESS services included because they were in danger of dropping 
out, needed extended learning time, needed to sustain present levels of performance, and needed 
to improve self-esteem. Thus, student performance and success in school were clearly the 
primary reasons that students received ESS services. 

Interviews with parents of students participating in ESS confirmed this finding, given 
that 22% of the 49 parents who were interviewed indicated that their children were participating 
in the program because they needed to improve their grades in a specific subject and 7% needed 
to improve their grades in general. Sixteen percent of parents cited the students’ need for 
additional help, 9% attributed enrollment to getting caught up on homework, 7% needed help 
staying on task, and 30% indicated a variety of reasons (including falling behind in schoolwork, 
having to make up tests, because the child wanted to participate, and supplemental reasons). 
Overall, parents perceived the reasons for their children’s participation in ESS to be to improve 
student performance and decrease risk of failure. 

In addition, 19% of the ESS students interviewed indicated that they were participating in 
ESS because they needed to do better in school (7%), had failed a class (5%), needed to improve 
a specific grade (4%), or needed to graduate (3%). Nearly one third (31%) of these students 
participated because they volunteered, indicating a desire to improve grades, get extra help, or 
simply wanting to come to the program as specific reasons for their participation. 

b. How are students referred for ESS services and by whom? 

According to 87% of the 225 ESS teachers surveyed, teacher recommendations were a 
basis for selecting students for participation in ESS. Other means of selecting students for ESS 
included parent requests (56%), student requests (45%), standardized test scores (9%), and other 
(10%). Similarly, 82% of the non-ESS teachers surveyed indicated that selection was based on 
teacher recommendations, 44% indicated that it was based on parent requests, 33% chose student 
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requests, 9% chose standardized test scores, and 8% selected other. Responses to the parent 
questionnaire paralleled these data, with 53% indicating teacher recommendation, 32% 
indicating student self-referral, and 29% indicating that parents referred the students. 

ESS teachers’ (« = 98) interview responses to the question about student selection were 
generally consistent with responses on their surveys. Thus, 24% of ESS teachers stated that 
teacher recommendations were used to identify students, 3% stated that school counselors 
selected students, and 2% stated that administrators referred students to the program. In 
addition, 16% of the ESS teachers responded that struggling learners were identified as needing 
ESS, 8% stated that test scores were used to identify students in need of ESS, and 18% and 14% 
of ESS teachers said selection was based on parents referrals or requests and students’ self- 
referrals, respectively. 

Thirty-one percent of school ESS coordinators who were interviewed (« = 23) said that 
referrals were made by teachers, 20% said they were made by parents, 10% said they were made 
by students, and 22% stated that eligibility was based on a variety of things including need and 
recommendations from professional school staff. Further, 25% of the 18 district ESS 
coordinators who were interviewed indicated that eligibility for ESS was determined by 
following district guidelines. In addition, 8% of the district level ESS coordinators interviewed 
said that the decisions regarding eligibility were made at the school level. Finally, when asked if 
there was a formal method of referral, 40% of the district level ESS coordinators interviewed 
indicated that teachers initiated referrals to which parents later consented, 10% said that 
community awareness was sought through advertising, and 10% indicated that other various 
methods were used (e.g., computer databases). 

c. Once the referral has taken place, how are individual student goals determined? 

Establishing individual students’ goals for the program appeared to rely heavily on 
parents and teachers, with some student participation. According to 30% of school ESS 
coordinators who were interviewed, the regular or classroom teacher, who was often also the 
referring teacher, determined the goals for students when he/she identified the students’ needs or 
reasons for referral. Another 12% of the coordinators said that the regular and ESS teacher set 
the goals together, and 6% of the coordinators said that the goals were stated on the referral 
forms. Only 24% of the ESS school coordinators said that students were involved in setting the 
goals for their participation, with 12% involving the regular teacher and student working 
together, 6% involving the students and ESS teacher working together, and 6% involving the 
students working alone. Moreover, although parents weren’t identified by the ESS coordinators 
as being part of the goal-setting process, 58% of the parents who were interviewed stated that 
they were indeed part of this process via collaboration with a teacher, as part of planning for 
college, by closely monitoring their children’s progress in school, and by working with their 
child at home. 

Regardless of the apparent lack of participation of students in the goal-setting process, 
only 35% of the ESS students interviewed said that they participated in ESS because someone 
else wanted them to do so. Thus, 17% of the students stated that their teachers suggested their 
involvement in ESS, and 1 8% indicated that one or both parents had encouraged it. Overall, it 
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seemed that students’ goals were heavily influenced by their parents and/or teachers but 65% of 
the students apparently accepted these goals as their own and/or imderstood why they were 
expected to benefit from ESS. 



Evaluation Topic Two: Profiles of Students Receiving Services 

a. What are the grade levels of students receiving ESS? 

ESS students in the sample schools were asked to provide information about their grade 
levels. A total of 1,195 students replied to this query on the student survey. Nearly half (48%) of 
those responding indicated that they were in the high school grades: 5% in 9th, 14% in 10th, 
another 14% in 11th, and 15% in 12th. ESS students in the elementary grades constituted the 
next largest group (28%), with 3% in 1st, 6% in 2nd, 7% in 3rd, 6% in 4th, and another 6% in 
5th. Slightly fewer (25%) were in middle grades: 1 0% in 6th grade, 8% in 7th, and 7% in 8th. It 
is interesting to note the increases in ESS students in the 6th grade and in the 10th grade; these 
increases perhaps indicate that students transitioning from the elementary and junior high schools 
may require the additional academic support offered via ESS. 

While nearly half of the students who returned surveys were at the high school level, this 
does not imply that more high school youth were participating in ESS sessions, but rather reflects 
the larger size of the high schools; further, half of the summer visits were conducted at the high 
school level. When comparing the percentage of ESS students at a school to the total student 
enrollment, overall building-level percentages were similar for elementary, middle, and high 
schools — from 16% to 19% for all three levels. 

b. What subgroups (gender, race, etc.) are represented by students receiving ESS? 

Gender data were available for students who completed the student survey in the sample 
school sites. Of the 1,163 students who responded to this demographic item, 52% (« = 603) were 
male and 48% (« = 560) were female. 

When disaggregated by school level, it appeared that the percentages of boys and girls 
attending ESS programs in the sample sites remained fairly stable, with boys’ participation 
increasing slightly at the middle school level. Boys constituted 49% of attendees in elementary 
schools, and girls, 51%. Middle school boys accounted for 53% of participants, and girls 
accounted for 47%. Likewise, 53% of ESS attendees were boys at the high school level, and 47% 
were girls. 

Several interesting discrepancies were apparent at the grade levels, however. For 
instance, in the 1 st grade, only 40% of students receiving ESS services at sample schools were 
boys. This overrepresentation of girls is anomalous. Dramatically, for example, 69% of ESS 
participants at the 9th grade level were boys, and only 31% were girls. (It should be noted that 
the numbers of students in these two grades are smaller than the numbers in the other grades 
represented.) Boys were also represented more than girls in ESS programs at sample sites in the 
7th (54%), 8th (57%), and 10th (54%) grades. 
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c. What are the achievement levels, as measured by grades and proficiency levels, of the 
students receiving ESS? 

According to 95% of the 151 district coordinators who replied to the survey mailed to all 
Kentucky district coordinators, most students were referred to the program because they were in 
danger of failing. Ninety-five percent also reported that a common reason why students were 
recommended for ESS was to improve their academic performance. 

On the other hand, of the 837 school ESS coordinators who returned their surveys, only 
76% indicated that students were referred to the program because they were in danger of failing. 
This represents a difference of 19 percentage points between the reports of district and school 
coordinators. Nonetheless, school coordinators (92%) tended to report, similarly to district 
coordinators, that students were recommended to the program to improve their academic 
performance. 

Data gathered during site visits suggested that students referred to ESS were at some risk 
for failure, but not to the degree suggested by respondents to the district and school coordinator 
surveys. Of the 18 district coordinators interviewed during AEL site visits, for example, fewer 
than half (45%) reported that students in their districts were recommended for program 
participation because they were not succeeding academically. Similarly, only 43% of the 23 
school coordinators interviewed during AEL site visits thought that students were referred to 
ESS because their performance was deficient (although an additional 6% reported that students 
were referred because they were not achieving to their potential). 

Interestingly, ESS teachers interviewed during AEL site visits reported much more 
frequently that students were referred because they were struggling academically. ESS teachers 
were asked “What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program in your district?” Of the 98 ESS teachers who replied to this 
question, 23% reported that the program was used to assist struggling learners, 13% to improve 
student performance, another 13% to prevent failure or attrition, 7% to improve performance in 
specific academic subjects, and 4% to improve test scores. ESS teachers, thus, provided a 
different perspective on ESS students’ performance than did coordinators. 

The 225 ESS teachers who completed an ESS teacher survey tended to corroborate this 
view of student performance. Asked to select fi"om a list the most common reasons students 
receive ESS services, 84% indicated that they were to improve their academic achievement, and 
three fourths (75%) noted that students were in danger of failing. In addition, 1 1% reported that 
ESS students were in danger of dropping out of school. 

Non-ESS teachers likewise thought that ESS students were referred to the program 
because they were performing poorly. Nearly three fourths (73%) of the total 297 non-ESS 
teachers responding to the survey reported that their students received ESS services because they 
were in danger of academic failure. Seventy-three percent also agreed that students were referred 
to ESS to improve their academic achievement. 
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d. What are the characteristics of students receiving ESS that put them at risk of 
dropping out of school — e.g, low achievement in school, poverty, single-parent homes, etc.? 

Data from the School/Program Description Form revealed some characteristics that may 
put students at risk of dropping out of school. For instance, when asked to describe any unique 
characteristics of their community, school, or student population, respondents most frequently 
noted general issues such as socioeconomic status of students (16%) and rurality (16%); at a 
more specific level, respondents noted a high incidence of eligibility for free or reduced-price 
meals (13%) and diverse locales (10%). 



Evaluation Topic Three: Profiles of ESS Programs and Their Implementation Patterns 

a. What are the major components of the ESS program? 

The major components of the Extended School Services program were identified by 
Kentucky educators in the mid-1990s as part of a research project completed by the Kentucky 
Institute for Education Research (KIER). Through a series of meetings, discussions, drafts, and 
reviews, KIER staff guided teams of Kentucky educators in developing of a series of “innovation 
component configuration maps” for all the major thrusts in the KERA law. Specifically, the 
component map developed for the ESS program was titled the Innovation Component 
Configuration Map for Extended School Services, or ICCM for short. The ICCM depicts the 
major components of the ESS program in the three broad areas of Student Eligibility, School- 
Level Program Design, and District- Wide ESS Programming Planning. 

There were unequal numbers of major program components within the three broad areas 
named above. In the Student Eligibility area, the components included referral guidelines, 
student selection, entry and exit process, and student assessment for eligibility. In the School- 
Level Program Design area, the components included school transformation planning, 
scheduling, staff selection, staffing patterns, instructional practices in ESS programs, organizing 
and grouping students, and instructional resources. Finally, within the District-Wide ESS 
Program Plaiming area, the major components included collaborative planning processes, 
program evaluation, fiscal management, and linkages with other KERA strands and other 
supporting programs. In sum, then, there were 15 major components in the ESS program, as 
derived from prior KIER research. 

Then, for each of the 15 major components of the ESS program, there was a series of 
possible implementation variations. There usually were three implementation variations for each 
component, but two components (instructional practices in ESS programs and program 
evaluation) had four possible variations. The component implementation variations were 
arranged from left to right on the page for each component and labeled as “Variation A” through 
“Variation C,” except for the two “Variation Ds.” These implementation variations were ordered 
from most ideal (Variation A) to least ideal (Variation D). The most ideal variation (A) was 
viewed by the ICCM developers as the implementation of that component that was the best that 
an ESS program could achieve. It follows, then, that the most ideal implementation of the ESS 
program would be one that was implementing all 15 major components at the “A” level. Of 
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course, in practical terms, it would be very difficult for any ESS program to achieve this high 
standard. To discover where schools in the state were with their implementation of the ESS 
major components, a variety of evaluation study data sources were inspected. 

b. How does the implementation of the ESS program components vary by stakeholder 
groups? 

During site visit interviews, district ESS coordinators were asked to describe their ESS 
services so as to solicit responses, of a general nature, about their implementation of the major 
components of the ESS program. Eighteen district ESS coordinators provided a total of 82 
responses to the question. These 82 responses were grouped into 1 7 categories, with the number 
of responses in each category ranging from 15 to 2. Comparing the response categories to the 
ESS major components and their implementation variations showed that eight different response 
categories, including 36 (44%) of the total responses, were related to some variation or other of 
the scheduling component in the School-Level Program Design area. For example, summer 
school was named 12 times (15%) by the district ESS coordinators, followed by afternoon 
programs, Saturday school, a.m. and p.m. services, intersession/breaks, before school, night 
classes, and off campus. The single category with the second-largest number (10 or 12%) of 
responses was labeled “monitor budget,” which relates most directly to variations of the fiscal 
management component in the District-Wide ESS Program Planning area. 

Also during site visits, school ESS coordinators were asked the same question about ESS 
services. Their responses were placed into 13 categories, ranging from 20 to 3 percent of the 
replies. It is interesting to note that seven of those categories can be collapsed and related to 
variations of the instmctional practices in ESS program components in the School-Level 
Program Design area. Totaling 63% of the school coordinators’ responses, the seven categories 
were subject/content areas, remediation/skills help, tutorial help, test preparation, 
writing/portfolio, homework, and higher-order skills. Another 6% of the responses on computer 
lab access were variations of implementation in the instmctional resources component in the 
same broad area. 

During the fall/winter and summer visits to ESS programs, 98 ESS teachers were 
interviewed and asked several questions about the implementation of key elements in their 
schools. As expected, these questions related to variations of implementation of major 
components in the area of School-Level Program Design. When asked about time of day for 
their ESS program, 52% of the teachers replied after school, 26% said summer school, 1 1% said 
morning before school, 6% replied Saturday, 3% said evening hours, and the remaining 3% said 
the times were flexible. ESS teachers were asked to describe the staffing of their programs, 
which related to the staffing patterns component in the ICCM. Thirty percent of the teachers’ 
responses fit into a category of elective/volunteer. The next highest category was miscellaneous, 
and contained 19% of the responses in a wide variety of statements. Teachers were asked to 
state the number of students per class, which also yielded information regarding the variations of 
the staffing patterns component. Thirty percent of the responses were in a category of 6-10 
students to 1 teacher, 17% were in 2-5 to 1, 17% in 11-15 to 1, 15% in 4-30 to 1. 
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During the site visits, the ESS teachers were asked a pair of questions related to the 
instructional practices in ESS programs component. When asked about the curriculum of their 
ESS program, the teachers’ responses fit into 10 categories, with responses ranging from 18% to 
3% of the total. Interestingly, the top 3 categories tied with 18% of the responses and included 
everything, same as regular classroom; reading; and math. When asked to describe the key 
elements of the instructional methods of their ESS programs, teachers provided a wide variety of 
responses fitting into 18 categories with 28% to 1% of the total. At 28%, the category with the 
most responses was individualized instruction, which was followed by the category of small 
groups at 13%. Finally, the ESS teachers were asked to describe their adaptations to student’s 
needs, which yielded variations of implementation for the organizing and grouping students 
component. Fully 35% of the teachers’ responses fit into the category of individualized 
instruction. Another 16% were in a category labeled learner needs (but not individualized). 
Then, the two categories of miscellaneous and appropriate level materials tied with 10% each. 

Also, as part of the site visit data collection, 49 parents of ESS students were interviewed. 
One question they were asked was “What are the best parts of ESS for your child?” The parents’ 
responses to this question provided some additional stakeholder information about the variations 
of implementations of the major components in the ICCM. Almost one third (32%) of the 
parents’ responses fit into a category labeled individual attention. At 15%, the second largest 
category dealt with the positive outcomes of the program on the academic standing of their 
children and was labeled improved academically. With 14% of the responses, the category of 
homework help was third largest, followed by the variety of activities in the ESS program at 
10%; and miscellaneous, also at 10%. 

During the 24 site visits to ESS programs, 109 students enrolled in the program were 
interviewed and asked several questions relating to the implementation variations of the major 
components. For example, related to the component of entry and exit process in the Student 
Eligibility area, students were asked how long they had been in ESS and what subjects they 
studied. Student responses to these two questions were often combined rather than separate, but 
four categories of responses were specific to the length of time in ESS. The category with the 
largest number of responses at 21% was called the first semester or first year. Then, 9% 
responded with second year. In the area of School-Level Program Design, students were asked if 
their ESS teacher was the same as their regular teacher. Fifty-six of the responses were in a 
category of not the same teacher, while 37% were in the same teacher category, and 5% said it 
was mixed (different for one subject, same for another). These different implementations relate 
to the staff selection and staffing patterns components in the ICCM. 

ESS students were asked “What do you learn about in ESS?” The responses helped to 
describe the variations of instructional practices and instructional resources in the School-Level 
Program Design area. Student responses to this question were categorized into 15 groups, with a 
few more of those groups naming a subject and the others naming an instructional practice or 
resource. Finally, students in ESS programs were asked several interview questions related to 
variations of implementation in instructional practices, organizing and grouping students, and 
instructional resources used in ESS. Students were asked to describe what they did in their ESS 
classes that was different or the same as their regular classes. Only 10% of their responses fit 
into the same category and just 4% of their responses were in the mixed (different and same) 
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category. The remaining 86% of the responses were spread over 13 difference categories. 
Students were asked if the ESS teachers did anything special to teach them in ESS class that their 
regular teachers didn’t and, if so, what. Student responses indicated that, indeed, ESS teachers 
did special things in ESS classes. More than three fourths of the responses (79%) were put into 
10 categories of different, special things, including more individual help (34%). Students also 
were asked if their ESS teachers told them how well they were doing. Student responses were 
sorted into 10 categories, with six of them having 70% of yes/positive responses and the 29% 
remaining responses in four categories. 

c. What are the patterns of implementation of the ESS components for the more 
effective ESS users and for the less effective ESS users? 

The patterns of implementation of the ESS components were determined through an 
analysis of the ICCM forms completed by the data collection teams at the conclusion of the 24 
site visits. There were 15 major ESS components on the ICCM; each had at least three possible 
variations (A-C) and two had four possible variations (A-D). For analysis purposes, each 
variation was assigned one point and the points per each completed ICCM were totaled. The 
total scores could range from a minimum of 15 to a maximum of 47; the higher the point value, 
the higher the level of ESS implementation toward the most ideal implementation of the most 
components. (Higher implementation scores often are called “high-fidelity implementations.”) 

Actual scores on the ICCM forms ranged from a low of 24 to a high of 45 on the 47-point 
maximum scale. There were 4 scores in the 40s, 15 in the 30s, and 5 in the 20s. To determine 
the patterns of implementation within these scores, the total implementation scores and the actual 
variation scores were examined simultaneously to discern similar groupings, or patterns. Four 
overlapping patterns of implementation emerged from this analysis: Pattern 1, scores of 24 to 31 
(« = 8); Pattern 2, scores of 33 to 38 (« = 7); Pattern 3, scores of 39 (« = 5); and Pattern 4, scores 
of 40 to 45 (« = 4). Given the nature of these patterns, a traditional graphic “map” was not 
generated for this analysis. 

The scores for Pattern 3 were very close to the scores for Pattern 4 and, likewise, the 
scores for some of Pattern 2 were close to the scores of Pattern 3. The distinguishing 
characteristic of the groups, then, rests in the details of the patterns of the implementations of the 
15 major ESS components. Put another way, although the total implementation scores were 
close, the patterns of implementation of the various components differed. For example. Pattern 4 
was the most effective — the ESS implementation with the most fidelity to the ICCM. Pattern 4 
implemented all components at variation A or B except for a single school that had all A and B 
variations plus two components with a C variation. In Pattern 2, the second-most effective ESS 
user group, all schools had implementation patterns consisting of all A and B variations and just 
one C variation. The implementation of components for Pattern 3, next-to-lowest effective users, 
consisted of some A and B variations plus either two or three C variations. Last, Pattern 1, the 
least effective ESS users, had implementation patterns either of some A and B variations plus 
four or more C variations or some A, B, C, and D variations. 

Another way to analyze the differences in the patterns of implementation of the ESS 
components was to look for differences within and across the two groups. That is, in addition to 
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the count of how many variations of each level each pattern had, an inspection was made of 
which variations were different both within and across Patterns 4 and 1. Within Pattern 4, all 
four ESS programs were judged to be at the B variation for the collaborative planning processes. 
Also within Pattern 4, three of the four programs were at variation B for the scheduling 
component. Within Pattern 1 (the lowest level of implemention), several trends were noted. For 
example, seven of the eight programs in Pattern 1 were judged to be at variations C or D for the 
program evaluation component. In fact, four of those were at variation D. 

Next, seven of the eight Pattern 1 programs were at variation C for the staff selection 
component, six of the eight were at variation C for the instructional resources component, and 
six of the eight were at variation C for the linkages with KERA strands and other supporting 
programs. Still within Pattern 1, all eight of the programs were evenly split between either 
variation A or B for the scheduling component and, also, six of the eight programs were judged 
at variation B for the organizing and grouping students component. Finally, when inspecting 
which variations differed across Patterns 4 and 1, the most that can be said is that there were no 
outstanding differences evident. No important differences were evident when comparing the 
Pattern 4 ICCMs to the Pattern 1 ICCMs. Of course, the total scores were very different, but the 
patterns that produced those total scores tended to be similar. Stated differently, which 
components were judged to be most ideal (more points) in one pattern seemed to be the same in 
the other pattern and vice versa for components judged to be less ideal (fewer points). 

To investigate further any possible differences across programs for Patterns 1 and 4, their 
School and Program Description Forms (SPDF) were reviewed. Unfortunately, one of the forms 
in Pattern 4 was missing (the school coordinator suffered an injury during the site visit and never 
submitted the completed form). The most interesting trend that emerged from the SPDF was that 
three of the four Pattern 4 schools were middle schools and the other was an elementary school. 
Also, for the three schools with completed forms, coordinators reported that students enrolled in 
the ESS program were just 9%, 10%, and 14% of the total school enrollment. For locale, two 
reported themselves suburban, and the other reported rural. They reported having six, eight, and 
nine ESS teachers — all of them also were regular classroom teachers. All were well-established 
programs, with two begun in 1990 and the other in 1994. The three Pattern 4 schools had after- 
school programs and just one school also offered weekend and summer programs. Two 
programs reported operating three days each week for about one hour each day. The other 
program reported operating five days a week for about four hours per day (this is the school that 
had the summer program). When describing the major components of their ESS programs and 
the current levels of implementation for each component, there were no trends appearing across 
the responses supplied by the three Pattern 4 schools, except that each offered some type of 
tutoring, remediation, or skills-building activities. Otherwise, the responses were rather 
idiosyncratic. 

The eight Pattern 1 (low implementing) school SPDFs were reviewed for possible trends 
across this group. With eight schools in the pattern, the possibilities for confirming trends were 
lessened and this was borne out in their responses to items on the form. There were three high, 
two middle, and three elementary schools in Pattern 1. Four of these schools reported being in 
rural locales, three in urban locales, and one in a suburban locale. The total student enrollment 
varied widely, as did the percentages of ESS students out of the total enrollment. These figures 
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ranged from 6% to 36% (with missing data for two schools). Interestingly, although the number 
of ESS teachers ranged from 4 to 25, the number for six of the eight Pattern 1 schools was 9 or 
less. And, similar to Pattern 4, all ESS teachers were regular classroom teachers. Also similar to 
Pattern 4, most of the ESS programs were well established, with just two of the eight starting as 
recently as 1996 or 1997. Seven of the eight schools had after-school programs, while the other 
was a summer program. Four of the eight also offered before-school ESS programs, five offered 
summer programs (other than the one already named), and four offered intersessions. 

To summarize the differences across the Pattern 4 (high implementers) and Pattern 1 (low 
implementors) SPDF sheets, the most interesting trends discovered were that three of the four 
Pattern 4 programs were in middle schools, and each of them had a rather small ESS program 
enrollment — ranging from 9% to 14% of the total school enrollment. 



Evaluation Topic Four: Services to Students Placed at Risk 

a. Are ESS programs serving students placed most at risk academically? 

One method to determine whether ESS is serving students placed most at risk 
academically is to examine how students are referred to ESS. Permitting multiple “paths” to 
ESS increases the likelihood that all children in need of ESS services will be identified. Teacher 
recommendation was used in all districts as a selection method. In two thirds of the districts, 
parent requests also served as a route into ESS. In 45% of the districts, students could request 
ESS services. In a third of the districts, standardized test scores were used to determine ESS 
eligibility. 

There are myriad reasons why a student might be considered at risk for failure, with the 
most obvious signal being poor performance in a subject. The main reasons students received 
ESS services were because they were in danger of failing (mentioned by 95% of district and 76% 
of school ESS coordinators), and/or needed to improve academic performance (noted by 95% of 
district and 92% of school ESS coordinators). 

More than 88% of all teachers, school coordinators, and district coordinators who were 
surveyed reported increased academic achievement as a main outcome of the ESS program. Of 
the 576 parents who answered the survey item, 58% said that their children had improved 
understanding of the subject material, 36% reported that their children were passing the subject, 
and 22% remarked that their children were passing the grade as a result of participation in ESS. 

b. Are ESS programs meeting the needs of students placed at risk academically? 

Questions were asked in several of the data collection activities to answer this evaluation 
question. The 49 parents of ESS students who were interviewed during site visits were queried 
about their perceptions of ESS effectiveness. When asked how their children were doing in 
school since participating in ESS, more than half (54%) responded that their children’s 
performance in school had improved, with comments such as “He has improved,” and the child 
had “brought grades up.” Seventeen percent of the parents believed that the instruction and 
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assistance their children received in ESS were helpful, saying, “They work better,” and that the 
child was “able to do homework.” The outcomes seemed not just related to specific content 
areas, but also included improved study skills or learning how to learn. 

A wider cross section of parents of ESS students was administered surveys. One 
question asked parents how their children’s performance in school had changed since they began 
participating in ESS. Parents were provided five answer choices, ranging from much better to 
much worse. Of the 565 parents who answered the question, 21% said they thought their 
children were performing much better in school since participating in ESS. More than half of the 
parents (57%) said their children were doing better in school. 

When asked during interviews what they thought their children had gained from the ESS 
program, 23% of the 49 parents commented that their children were giving school more attention 
because school was giving the children more attention, i.e., one-on-one instruction and tutoring. 
Better grades were a noticeable outcome mentioned by 15% of the parents interviewed. 
Fourteen percent of parents had witnessed increased self-esteem in their children. Ten percent of 
the parents believed their children’s participation in the ESS program had resulted in better 
homework skills (“She is better able to do her homework”); an additional 9% of parents had 
noticed their children having better general study skills and organizational skills. 

Both ESS and non-ESS teachers were asked in the teacher surveys what the most 
important ESS outcomes were for students. The teachers could provide more than one response. 
Almost all (95% of the ESS and 89% of the non-ESS teachers) reported enhanced academic 
achievement. Two thirds (65%) of the ESS teachers observed increased motivation on the part 
of students; in comparison, 38% of the non-ESS teachers noted increased motivation as an 
outcome. Sixty percent of the ESS teachers and 29% of the non-ESS teachers indicated that 
students had better self-esteem because of participation in ESS. 

District and school ESS coordinators were asked the same question about student 
outcomes. Almost all (99%) of the district coordinators stated enhanced academic achievement 
as an outcome of the program. Approximately two thirds (62%) of the district coordinators 
marked increased motivation as an outcome, and 48% reported increased student self-esteem. 
School coordinator responses were similar, with 98% noting enhanced academic achievement, 
70% reporting increased student motivation, and 56% noting an increase in student self-esteem. 

On the student survey was the statement, “I am a better student this year,” to which 
students either agreed or disagreed. More than three fourths of the 1,201 students (78%) 
responded that they were better students this year. Eighty-five percent of students agreed that the 
ESS program was helping them this year, and 86% stated that they asked for help in ESS when 
they needed it. Finally, when asked if they liked school, 67% responded affirmatively. 

One item on the parent survey asked parents to list the best features of the ESS program. 
The 461 parents who responded to this question provided a total of 522 discrete responses. 
Twenty-one percent of parents said that the best feature was the extra help the program provided 
to students. Helping students to better understand their schoolwork was a best feature noted by 
1 5% of the parents surveyed. 
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Most of the 49 parents (72%) responded positively when asked during interviews if they 
thought their children’s ESS teachers were helping the children to do their best. Parents made 
comments such as, “Teachers are wonderful,” “ESS teachers do help her to do her best,” and 
“[my child] speaks well of her.” ESS students were also asked during interviews whether and 
how their ESS teachers might be different from their regular classroom teachers. Thirty-four 
percent of students interviewed stated that the teacher(s) provided more individualized 
assistance. Nineteen percent responded that there was no difference between teachers. 

In the student survey was an item asking students what they liked best about the ESS 
program. The top five responses supported what students in interviews had said about their ESS 
teachers differing from their regular classroom teachers. Thirty-nine percent of students 
responded that they liked the tutoring and individualized instruction available in ESS. Fourteen 
percent appreciated having the extra learning time that ESS provides. 

c. Are ESS programs identifying and addressing those factors that place students at risk 
for failure? 

There are challenges in working with students placed at risk academically. A quarter of 
the ESS teachers who were surveyed commented that motivating students and getting students to 
attend ESS were hurdles they faced (“Motivating students while focusing on academic 
challenges”). Attendance and student motivation were challenges mentioned also by non-ESS 
teachers (22%), school ESS coordinators (19%), and district coordinators (12%). 

The emphasis at most schools appeared to be individualized instruction, or tutoring. This 
type of instruction was helping students who needed extra time and assistance to master material. 
Students were highly appreciative of the extra attention from teachers that ESS provided. 
Eighty-five percent of those surveyed indicated that ESS was helping them that school year, and 
39% of ESS students who were interviewed commented that the individual tutoring was helping 
them. The additional time to do make-up work or redo work or tests on which the student 
received a poor grade also were mentioned by 3 1 % of the students who responded to the survey 
as the aspect they liked best about the ESS program. 



Evaluation Topic Five: ESS Implementation Patterns and Outcomes 

a: How does the fidelity of ESS implementation correlate with academic index scores? 

As noted under evaluation question three, four distinct patterns of ESS implementation 
emerged from the ICCM instrument. These implementation scores were used to determine 
Pearson correlation values with academic index scores from the 2000-2001 Kentucky Core 
Content Tests (Kentucky Department of Education, 2002) for the following subjects: reading, 
science, mathematics, writing, social studies, arts/humanities, and practical living/vocational science. 
Table 4 presents correlation values of ESS implementation vsdth each of the above subjects. 

As can be seen in Table 4, correlation values between the ICCM total implementation 
scores and each of the school-level variables were very small, indicating a lack of relationship 
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between them. None was significant, which was not unexpected given the small sample size of 
u 24 implementation scores. 

b: How does the fidelity of ESS implementation correlate with school-level variables such 
as retention, discipline, attendance, etc. ? 

Again utilizing the ICCM total implementation scores, Pearson correlations were 
generated for a number of school-level variables, including attendance rate, retention rate, 
average years of teaching experience, the number of drug/weapon/assault incidents, the spending 
amount per student, and the number of parent volunteer hours (see Table 4). Again, these data 
were culled from the school report cards provided on the Internet (Kentucky Department of 
Education, 2002). As noted above, correlation values between the ICCM total implementation 
scores and each of the school-level variables were very small, indicating a lack of relationship 
between them. None of the correlations was statistically significant. 



Table 4: ICCM Implementation Score Correlations With School Variables 



School Variables 


Pearson Correlation 
with ICCM Total 
Implementation Score 


KCCT* Reading - Academic Index 


-.044 


KCCT Science - Academic Index 


-.067 


KCCT Mathematics - Academic Index 


-.010 


KCCT Writing - Academic Index 


.039 


KCCT Social Studies - Academic Index 


-.234 


KCCT Arts/Humanities - Academic Index 


-.018 


KCCT PLWS - Academic Index 


-.015 


Average Years of Teaching Experience 


.021 


Number of Parent Volunteer Hours 


.077 


Spending Amount per Student 


.233 


Attendance Rate 


.147 


Retention Rate 


-.028 


Number of Drug, Weapon, or Assault Incidents 


.029 



*Kentucky Core Content Test 



Note : ICCM scores could range from 15 to 47; the higher the score, the higher the level of ESS 
implementation; actual scores ranged from 24 to 45. 

c. How does the fidelity of ESS implementation distinguish between schools with 
minimum and maximum achievement gaps? 

Each of the site visit schools was identified by KDE staff as having either a minimum or 
maximum achievement gap in overall academic index scores between White and minority 
students. To determine whether the ICCM implementation score could differentiate between 
these two classifications, an independent t test was conducted. The mean ICCM implementation 
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score for the 13 schools classified as minimum gap was 36.54 (standard deviation of 5.25); the 
score for the 11 maximum gap schools was 32.45 (standard deviation of 5.97); the mean 
difference was 4.08. With a t value of 1.78, and a significance of .088, this difference 
approached but did not reach statistical significance at the .05 level. 

d: What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within the classroom observations? 

For the ESS classroom observations, eight-minute segments were averaged into one set 
of scores per observation, then aggregated and classified by school into the four patterns of ESS 
implementation (lowest to highest levels of implementation) as defined by the ICCM instrument. 
For the classroom snapshot of the observation, students were coded during a one-minute 
observation as being on task, off task, out of the room, or waiting. One-way analyses of variance 
(ANOVAs) were conducted to determine whether statistically significant differences occurred 
among the mean number of students in each of the activities by implementation pattern. No 
significant differences were found between any of the pattern groups. For the number of on-task 
students, means ranged from 8 to 10 across patterns (standard deviations of 3 to 4); for the other 
three activities, means were all below 1 (standard deviations of 0 to 1). 

Then, attention was given to the target student segment of the ESS classroom 
observations. Again, eight-minute segments were averaged into one set of scores per 
observation, then were aggregated and classified by school into the four patterns of ESS 
implementation. The 27 individual activities that a target student could be involved in were 
grouped into the four main categories of teacher-led, management/organization, student-led, and 
off-task. One-way ANOVAs were conducted to determine whether statistically significant 
differences occurred among the mean number of minutes spent in each of the categories by 
implementation pattern. While no significant differences were found between any of the pattern 
groups. Figure 2 does show fluctuations across patterns. 

Time spent in teacher-led activities ranged from an average of 21 minutes in Pattern 4 
(highest level of ESS implementation) to 30 minutes in Pattern 3 (second highest level of 
implementation), with standard deviations of 3.46 and 16.44, respectively. Much less time was 
devoted to management/organization, as seen by a high of 5 minutes in Pattern 4 (standard 
deviation of 3.30) to a low of 3 minutes for the other three patterns (standard deviations ranging 
from 1.25 to 1.94). Time spent in student-led activities occurred most often in Pattern 4 with a 
mean of 25 minutes (standard deviation of 7.15), and least often for 16 minutes in Pattern 3 
(standard deviation of 8.72). Finally, less time was devoted to off-task activities, with 8 minutes 
in Pattern 1 (lowest level of implementation, standard deviation of 7.34) and 4 minutes in 
Patterns 2 and 4 (standard deviations of 2.33 and 3.93). 

e. What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within selected data measures? 

The following analyses are further investigations of what factors might be underlying the 
four different patterns of ESS implementation. In particular, attention was given to inspecting 
forces that led to successful ESS implementation and barriers that hindered implementation. 
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Figure 2: Observation Data: Percent of Time Spent in Main Categories 
by ICCM Implementation Pattern 



Available data from the 24 school coordinators from the statewide survey administration were 
utilized for this analysis. Further, data from ESS teachers’ surveys during the 24 site visits were 
aggregated by school and classified into four patterns of ESS implementation. The seven forces 
included (1) clear support or mandate from district or other political actions, (2) clear support from 
parents or community, (3) additional financial support, (4) excellent staff development and follow- 
up, (5) excellent relationships among staff, (6) outstanding administration, and (7) other. 

All ESS teachers identified five of the seven forces as aiding their ESS implementation: 
district support, community support, financial support, staff relationships, and administration. 
Interestingly, district support was the only force that teachers within schools in Pattern 4 (highest 
level of implementation) selected more often than teachers at other schools; Pattern 4 had 50%, 
Pattern 1 (lowest level of implementation) had 38%. For the other four forces, ESS teachers in 
Pattern 1 showed higher percentages than Pattern 4 (differences of at least 25%), indicating a 
greater perception that these forces were prevalent in their schools. In comparison, school 
coordinators identified the above forces, along with staff development. Data showed two forces 
(district support and staff relationships) with large differences between Patterns 1 and 4 (75% 
difference in district support and 25% in staff relationships). Generally speaking, coordinator 
responses were more indicative of specific forces aiding ESS implementation than the ESS 
teachers. Within the 20 groupings (five forces identified by both teachers and coordinators by four 
patterns), 13 were 60% or more for school coordinators, compared to 7 for teachers. 
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Teachers and school coordinators identified 10 problems or obstacles were encountered 
in their schools as they implemented their ESS programs. These obstacles included (1) problems 
with state or district regulations; (2) opposition or demands from key district, school, or other 
staff; (3) opposition or demands from parents or community; (4) problems with teacher unions; 
(5) inadequate financial support; (6) inadequate preparation of teachers or other school staff; (7) 
problematic relationships among school staff; (8) student transportation; (9) opposition or 
demands from students; and (10) other. 

ESS teachers selected only two problems — inadequate financial support and student transpor- 
tation. A fourth of teachers within Patterns 1 and 4 each selected inadequate financial support; no 
Pattern 2 or 3 teachers selected this problem. All teachers indicated student transportation was a 
problem; however. Patterns 1 and 4 were equal at 25%. In comparison, school coordinators selected 
four problem areas: inadequate financial support, student transportation, student opposition, and 
other. For Pattern 4, 100% of the coordinators indicated inadequate financial support, compared to 
13% of the Pattern 1 coordinators. Similarly, more Pattern 4 coordinators indicated some other 
reason was causing problems (33%), compared to 13% of the Pattern 1 coordinators. For student 
transportation, the trend reversed, with 75% of the Pattern 1 coordinators indicating this was a 
problem, compared to 33% of the Pattern 4 coordinators. Similarly, more Pattern 1 coordinators 
indicated student opposition (50%), compared to Pattern 4 coordinators (33%). Generally speaking, 
school coordinators were more indicative of specific problems hindering implementation than the 
teachers. Within the eight groupings (two problems identified by both teachers and coordinators by 
four patterns), two were 60% or more for school coordinators, compared to none for teachers. 

Next, data resulting from the administration of the AEL CSIQ instrument were 
aggregated by school and classified by ESS implementation pattern for the site visit schools to 
determine whether statistically significant differences in scale scores occurred between patterns. 
One-way ANOVAs were generated for each of the six scales by the four patterns of ESS 
implementation. One statistically significant difference was found in the School/Family/ 
Community Connections scale (F(3,20) = 3.39, p < .05). Tukey’s HSD was used to pinpoint 
which patterns differed significantly and revealed that only Pattern 4 (highest implementation 
level) differed from Pattern 3 (second highest implementation level). Pattern 3 had a mean score 
of 50.57 (standard deviation of 1.97), compared to a mean for Pattern 4 of 43.32 (standard 
deviation of 4.15) for a mean difference of 7.25. There is a large effect size associated with this 
difference (-2.23), indicating that there is not only a statistically significant difference, but also a 
meaningful one in a practical sense. While only one significant difference was found. Figure 3 
does show slight fluctuations among all six of the scales. 

Finally, Pearson correlations were generated for the six mean AEL CSIQ scale scores of 
the site visit schools with the ICCM total implementation score. The correlations follow: 
Learning Culture, .137; School/Family/Community Connections, -.070; Shared Leadership, .046; 
Shared Goals for Learning, .037; Purposeful Student Assessment, -.031; and Effective Teaching, 
.211. These correlations were all very small, indicating a lack of relationship between the scales 
and the implementation score. None of the correlations was statistically significant. 
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Figure 3: Mean AEL CSIQ Scale Scores by ICCM Implementation Pattern 
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CONCLUSIONS 



A number of conclusions can be drawn from the findings presented within this 
comprehensive evaluation of the statewide Kentucky Extended School Services program. These 
are organized by nine topical areas. 



Student Demographics 

• In terms of the proportion of ESS enrollment to student enrollment, ESS participation is 
fairly equal across elementary, middle, and high school building levels; however, 
participation varies widely at the individual school level. 

• The participation of boys and girls in ESS is roughly equivalent, particularly at the 
elementary level. However, fewer females participate in the program at the middle and 
secondary levels. This warrants further investigation to determine whether middle and 
high school girls need fewer ESS services or if they are simply less interested than boys 
in ESS participation. 

• Students attending ESS programs are characterized by coming from poorer areas (rural 
and iimer city), which lack resources. These circumstances place students at risk of 
academic failure and dropping out of school. 



Adherence to Intended Goals 

• Generally, students are referred to ESS because they are not performing well 
academically and may be in danger of failing. Other reasons noted were to extend 
students’ learning time, sustain students’ current levels of performance, or improve 
students’ self-esteem. Some students taking advantage of ESS services do so because 
they are in jeopardy of failing at least one class or subject. Thus the achievement of most 
ESS students is depressed when they first begin participating in the program. 

• There is a great deal of consistency among the perceptions of coordinators, teachers, and 
parents as to how students are referred to ESS; the majority believe that students are 
referred most often by classroom teachers. However, students report that they most often 
self-select into the program. It may be that students are taking credit for self-selection by 
agreeing to participate in this voluntary program after a teacher or parent has made the 
suggestion. Either approach seems to allow enough flexibility for the intended 
population to become involved with the program. 

• The students’ regular teachers, ESS school coordinators, and ESS teachers most often 
determine individual student goals, with parents and students themselves being involved 
to a lesser extent. Thus students’ goals appear to be heavily influenced by their teachers. 
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yet the majority of students adopt these goals as their own and appear to understand why 
they are expected to benefit from participation in the ESS program. 

• There is congruence among perceptions of the intended and actual outcomes of the ESS 
program. All stakeholders agree that the ESS program is helping students increase their 
academic achievement, pass courses and grades, and decrease school failure. 

• Given the main reasons for referral, and the outcomes perceived by respondents, it is 
evident that the ESS program is operating within its framework and addressing the main 
goals it is intended to accomplish. 



Classroom Instruction 

• ESS and regular classrooms differ on two major dimensions: quality of instruction and 
appropriate level of instruction. Quality of instruction is better in regular classrooms, but 
instructional level is more often appropriate in ESS classrooms. 

• ESS classrooms tend to engage in student-led activities, often involving independent 
seatwork and pair seatwork. Thus a “typical” ESS classroom appears to be one in which 
students work independently on homework and/or make-up tests, receiving 
individualized instruction as needed. One strength of the ESS classroom arrangement is 
that students are receiving the one-on-one tutoring they need and have the opportunity to 
have concepts not mastered retaught to them. 

• While computers are almost universally available in both ESS and regular classrooms, 
very limited use was made of this resource. However, the environmental checklist did 
not differentiate between one or multiple computers, so in classrooms with only a single 
computer, usage may be restricted to teacher purposes. 



Student Outcomes 

• The ESS program appears to be having an impact on student performance. Nearly all 
teachers and coordinators indicate that participation in ESS has led to increased academic 
achievement. Further, parents report increased understanding of subject material by their 
children, that their children are passing a particular subject, or that their children are now 
doing better in school. 

• Parents and students also report improved study skills and increased motivation to learn 
as a result of participation in ESS. Students appreciate having opportunities to make up 
or retake tests. This flexibility for students who either missed a test or performed poorly 
on a test indicates that value is placed on allowing students the opportunity to show what 
they have learned. 
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y •For many students, ESS provides a time to receive individualized instruction, to learn 

y study skills, and to have learning reinforced through the use of games, visual aids, 

practice, additional time, and incentives. 



Program Strengths 

• The major strengths of the ESS program focus on processes for its implementation and 
outcomes resulting from that implementation. For instance, process-linked supports 
include targeting students as early as possible, dedicated staff, student transportation, 
collaboration between teachers and coordinators, flexible scheduling, low teacher/ 
student ratio, and individualized instruction. 

• There is a high degree of continuity between coordinators’ and teachers’ beliefs about 
key forces that help the ESS programs to succeed. The most critical components for 
successful implementation are strong district- and building-level support. Other critical 
components for implementation success are collaboration and relationships among staff, 
parent or community support, staff development, and financing. 

• Coordinators’ responses confirm that there are numerous successful programs operating 
in many schools in Kentucky. One particular reason given for success was the use of 
innovative and creative ESS methods. 

• One unique strength of the ESS program is its fluidity and flexibility. Student mobility is 
high throughout the program. As a particular problem arises, ESS allows for an 
immediate intervention that focuses on a specific need that can be addressed before it 
becomes chronic and long term. The program does not rely solely on the results of 
annual standardized test scores, which would slow down the process of identification, 
referral, and enrollment. 



Barriers to Maximum Success 

• A variety of topics are viewed both as weaknesses and as strengths, depending on their 
presence or absence. These include student transportation, funding, staff development, 
parental communication, staffing, and student motivation. This suggests that when these 
factors are in place and sufficient, they provide a strong foundation for successful ESS 
implementation. Conversely, the absence or insufficiency of these factors is detrimental 
to maximizing the potential of an ESS program. These issues are more fully discussed in 
the context of weaknesses so that administrators and policymakers can see the 
explanatory comments related to each. 

• Student transportation is a major problem for some schools. The decision to use ESS 
funds to provide public transportation for students is determined by individual school 
and/or district policies. Because the majority of the ESS services offered during the 
regular school year occur after normal school hours, if bus service is not provided then 
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parents must make transportation arrangements for their children. With the combination 
of parental work schedules, a potential lack of transportation for lower-income families, 
and the distance involved for more rural communities, this factor could seriously deter 
participation of some students who might be most in need of such academic services. 

• Staff development related to ESS now seems to be nonexistent, inadequate, or distributed 
unevenly between teachers and coordinators. This may be more problematic for newer 
staff members who are initially becoming involved with ESS and who are not familiar 
with its related philosophies and guidelines, especially since the ESS summer conference 
was discontinued. Further, there is some lack of agreement among school coordinators, 
teachers, parents, and students as to the exact intent and nature of the ESS program. 

• One discrepancy noted among respondent groups involves communication, especially 
with parents. While ESS teachers believe they meet with parents on an as-needed basis, 
parents note that communication with the teacher about their children’s progress is a 
major problem and that they often are not aware of ESS goals. 

• There seems to be some degree of misunderstanding regarding the emphasis on core 
subjects taught in ESS sessions. District coordinators’ perceptions seem to be most 
closely aligned with the parameters of the ESS policy and regulations. 

• Most of the respondents believe that the current number of teachers involved in ESS is 
inadequate for the number of students. Related to this topic is the reported difficulty 
associated with recruiting, hiring, and retaining a sufficient number of interested teachers 
with appropriate content knowledge and relevant skills for working individually with 
students in the ESS environment. 

• Student motivation is a relevant issue for encouraging participation in the ESS program. 
Although some students are not motivated enough to participate, those who do participate 
tend to become more interested and to improve their academic performance as a result. 
Moreover, students consider the use of alternative, “fun” instructional strategies in the 
ESS classrooms as more engaging. 

• Finally, there is consent among the coordinators and teachers that additional funding is 
necessary to adequately support full implementation of the ESS program. Addressing 
several of the weaknesses noted above would require an increased level of funding to 
provide consistent student transportation, staff development, expanded services in terms 
of hours and/or subjects, and a reduction of the student/teacher ratio. 



Program Fidelity 

• The ESS programs are performing satisfactorily in terms of implementing the majority of 
the 15 major components of the statewide program. The following four components 
seem to be implemented least satisfactorily: staff selection, instructional resources, 

collaborative planning processes, and program evaluation. 
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There are four types of implementation of ESS programs in terms of their fidelity in 
operating the 15 major components of the program. That is, there are four levels of 
implementation of the ESS program, ranging from high-fidelity implementers to low- 
fidelity implementers. However, these patterns of implementation are very similar across 
the four groups; the main differences are in the levels of implementation of each 
component, as opposed to the differences across the components. Three of the four high 
implementation schools are middle schools with small ESS programs in terms of the 
number of involved students and teachers. In other words, the high-fidelity 
implementation is more an artifact of program scale and building level rather than 
discrete differences in implementation. 



Patterns of Implementation 

• Although there seem to be no discemable operational differences in the four levels of 
implementation, there are some differences in associated measures when compared by 
implementation pattern. The high implementation group consistently spent less time on 
teacher-led activities and more time on student-led instructional activities than any of the 
remaining three groups. 

• When looking at implementation patterns with other data measures utilized in this 

comprehensive evaluation, one other conclusion can be drawn: All the ESS school 

coordinators in the high implementation group pinpointed inadequate financial support. 



Overall 

• One of the most striking conclusions from this comprehensive evaluation of the statewide 
Kentucky Extended School Services program is the marked consistency and high degree 
of corroboration both within and among respondent perceptions and data collector 
observations. 

• Overall, it is concluded that the ESS program is positively perceived by involved 
stakeholders and has been proven to help address the needs of students who are at risk 
academically. However, several areas have been identified in which improvements could 
be made for a more successful implementation of the statewide program. 
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RECOMMENDATIONS 



Based on the findings and conclusions of the statewide Kentucky Extended School 
Services evaluation, a number of specific recommendations are offered for KDE staffs review 
and reflection. 

• Some thought should be given to making scheduling adjustments to the after-school ESS 
programs, such as expanding hours of operation or simply staggering scheduled times 
within a week. The demand for additional time in ESS will need to be weighed against 
the possibility of further exacerbating the conflict between ESS sessions and 
extracurricular activities and/or part-time jobs. 

• ESS staff should encourage/facilitate more involvement of parents and students in setting 
goals for individual students. This would help to improve communication between the 
home and school and to ensure that all involved parties share similar goals for individual 
students’ learning — further increasing the likelihood that these goals will be uniformly 
sought, supported, and achieved. In addition, continued communication with parents 
about their children’s progress should be a routine part of ESS program operation. 

• Professional development opportunities should be provided to ESS coordinators and 
teaching staff in the areas of staff selection, instructional resources, collaborative 
planning processes, individualized instruction, mentoring/tutoring, and program 
evaluation. The specific format for these professional development opportunities could 
vary from workshop sessions at a central site or decentralized sites to online. Internet- 
based courses. Whatever delivery method is selected, professional development in these 
four areas is needed by most ESS program staff in the state. 

• School-level ESS staff should carefully consider the scale of the program as they plan, 
deliver, and evaluate their programs to improve the level of implementation. Rather than 
resorting to downsizing, ESS staff need to assess how thoroughly and effectively they 
have implemented the 15 major program components and develop an action plan for 
improving those areas identified as being low or poor. 

• KDE staff and state board of education members should collaborate to identify possible 
solutions to transportation issues. Solutions might include working closely with 
transportation staff, investigating alternative funding formulas such as using non-ESS 
monies for transportation expenses and/or seeking additional funds specifically for 
transportation. 

• Some thought should be given to exploring ways to overcome the teacher staffing issue. 
For example, KDE staff could identify those districts experiencing ESS teacher 
recruitment problems and work with them to develop solutions. If the problem is teacher 
pay for ESS sessions and state or local regulations that prevent increasing teacher 
salaries, perhaps KDE staff could be instrumental in finding ways to overcome those 
barriers, such as seeking waivers for current rules or regulations. 
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The possibility of developing an incentive program for ESS teachers that would generate 
opportunities for recognition of their efforts should be investigated. For example, an ESS 
Teacher of the Year award program might be designed and implemented. The idea is to 
offer a significant award and possibly a financial reward based on state-established 
criteria. The award, which could be regional or statewide, may help draw teachers 
previously uninterested in participating in the ESS program. 

The summer conference for ESS coordinators and teachers should be re-instituted. This 
conference provides an excellent opportunity for numerous professional development 
sessions for ESS coordinators, teachers, and staff from any district. Also, the 
opportunities to share ESS program information, successes, and solutions to common 
problems would be greater at a large conference. The added value would be that 
professional associations and networking about ESS across the state would likely evolve 
from such a conference; for example, a statewide organization of ESS professionals. 

It should be clearly communicated to all stakeholders that the ESS program, as 
implemented under current laws and regulations, is not designed to be an enrichment 
program. A clear understanding of the specific nature and purpose of the statewide 
program may help avoid efforts to shift its focus from struggling learners to all students. 

The current mechanism of categorical funding for the individual ESS programs should be 
maintained. Nearly all district and school respondents agreed this system worked well 
and felt that funds were distributed equitably. 

KDE and local ESS school staff should investigate ways to recruit at-risk and hard-to- 
reach students. Identified successful methods could be included in the best practices 
resource described below. Schools or districts could apply for grant money to fund 
focused, intensive efforts to increase students' awareness of and interest in the ESS 
program. Other possibilities include modifying current ESS activities to make them more 
fun for students by introducing creative, innovative instructional strategies to better 
capture students' interest, or experimenting with an incentive system to provide more 
extrinsic, short-term rewards to give students a sense of accomplishment during their 
participation in the ESS program (in addition to the intrinsic, long-term goal of increasing 
their academic achievement). 

KDE staff should formalize and fund the process for obtaining ESS “best practices” and 
develop a resource tool that would be available to all ESS staff. ESS staff in one or more 
districts could be financially compensated for spearheading the initiative and gathering 
submissions from all ESS programs. The final product could be in print or electronic 
format and would be a compendium of innovative and creative ESS programs. It could 
also include a segment on student motivation, as mentioned earlier. We understand that 
such an effort is currently under way, but statewide coordinators indicated limited 
awareness of this undertaking. Therefore, at the very least, KDE staff should increase the 
visibility and potential utility of such a tool for the ESS program statewide. One potential 
resource is the Promising Practices in Afterschool (PPAS) Web site, which provides 
detailed descriptions of promising practices nationwide (see www.afterschool.org). 
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